gh-131656: Solved the cross-interpreter support of multiprocessing module
#131657
Conversation
Cross-interpreter support for multiprocessing module
|
All commit authors signed the Contributor License Agreement. |
|
Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool. If this change has little impact on Python users, wait for a maintainer to apply the |
pypy66
changed the title
pypy66
changed the title
multiprocessing module
| for path in data['sys_path']: | ||
| if path in sys.path: | ||
| sys_path.remove(path) | ||
| sys.path.extend(sys_path) # For cross-interpreter support |
There was a problem hiding this comment.
this should not be needed with the user supplied sys_path manipulation function in the parent. that can already know what the entire sys.path of the set_executable interpreter should be and pre-populate the value being sent to the child. no in child filtering or extend needed.
There was a problem hiding this comment.
@gpshead But how can the parent process detect the appropriate sys.path of the child since the child may has its own configs (e.g.HKEY_CURRENT_USER\SOFTWARE\Python\PythonCore\3.12\PythonPath on Windows) and its own implementation of how to read these configs that differs from the parent.
Firstly, I've decided to simply replace the base path. For instance, D:\Python313\Lib is simply replaced with D:\Python312\Lib, and D:\Python313\DLLs is replaced with D:\Python312\DLLs,
but if the Python 3.12 is installed inside Cygwin and its sys.path is D:\cygwin\usr\local\lib\python3.12, there is no way to detect it.
So, I suppose that merging the paths is a better and more elegant solution:
# In the parent process
sys_path=[] # The paths being sent to the child
for path in sys.path:
if sys.base_prefix not in os.path.abspath(path): # Filter out the path that is not related to the parent (e.g. current working directory) by simply using `sys.base_prefix`
sys_path.append(path)
...In the child:
for path in data['sys_path']:
if path in sys.path:
sys_path.remove(path) # remove duplicate paths
sys.path.extend(sys_path)However, my code using sys.base_prefix may be not a elegant way since you cannot use not in in Linux. Could you suggest a better way, please?
|
A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated. Once you have made the requested changes, please leave a comment on this pull request containing the phrase |
The parent could run the child interpreter once when setting all of this up to get it's sys.path. |
That it a good idea while it has a slight performance overhead, I'll do it. |
|
I've inadvertently deleted my CPython fork, so I'm using my another account @qfcy to re-create this pull request, sorry. |
2 participants
The issue #131656 could be solved by slightly modifying the
prepareandget_preparation_datafunctions frommultiprocessing.spawnmodule, which is responsible for exchanging Python runtime states includingsys.path.After modifying, my example code in #131656 successfully runs:
multiprocessingmodule #131656